7th Lattice QFT Summer School



Institute of Modern Physics, Huizhou Campus

2024-08-07

\gdef\dst{\displaystyle} \gdef\pd#1#2{\frac{\partial{#1}}{\partial{#2}}} \gdef\ppd#1#2{\frac{\partial^2{#1}}{\partial{#2}^2}} \gdef\vnbl{\vec{\nabla}} \gdef\vE{\vec{E}} \gdef\vB{\vec{B}} \gdef\vr{\vec{r}} \gdef\vA{\vec{A}} \gdef\vJ{\vec{J}} \gdef\vF{\vec{F}} \gdef\dvS{d\vec{S}} \gdef\dvl{d\vec{l}} \gdef\dlt{\delta} \gdef\dlt3{\delta^{(3)}} \gdef\eps{\epsilon} \gdef\d3r{d^3\vec{r}} \gdef\tnsrr#1{\tnsr{#1}} \gdef\tnsr#1{\overset{\mathrlap{\mkern-4.5mu\boldsymbol{\rightharpoonup}}\boldsymbol{\rightharpoonup}}{#1}} \gdef\vi{\vec{i}} \gdef\vj{\vec{j}} \gdef\vk{\vec{k}} \gdef\vr{\vec{r}} \gdef\ve{\vec{e}} \gdef\grad#1{\vec{\nabla}\;{#1}} \gdef\curl#1{\vec{\nabla}\!\!\times\!\!{#1}} \gdef\div#1{\vec{\nabla}\!\cdot\!{#1}} \gdef\({\left(} \gdef\){\right)} \gdef\[{\left[} \gdef\]{\right]} \gdef\pd#1#2{\frac{\partial{#2}}{\partial{#1}}} \gdef\ppd#1#2{\frac{\partial}{\partial{#1}}{#2}} \gdef\pdd#1#2{\frac{\partial^2{#2}}{\partial{#1}^2}} \gdef\ppdd#1#2{\frac{\partial}{\partial{#1}^2}{#2}^2} \gdef\box#1#2{\boxed{#1}_{~(#2)}} \gdef\grad#1{\vec{\nabla}{#1}} \gdef\div#1{\vec{\nabla}\cdot{#1}} \gdef\cur#1{\vec{\nabla}\times{#1}}

image

s1 SSH remote Login

s2 Linux basic operations

s3 Visual Studio Code

s4 Python

  • virtual environment

  • useful packages numpy, matplotlib, gvar, lsqfit

s5 slurm and environment modules

image

s1

SSH remote Login

image

s1 SSH remote Login

Linux and MacOS

  • In terminal
    ssh -p port user@address

    • port [optional] : port number
    • user: username
    • address: address of the server

    e.g ssh -p 10023 USR@222.200.137.16

Windows

  • Powershell  
  • Windows terminal
  • Windows Subsystem for Linux (WSL)

image

Use ssh config file

  • Open or create config file

    • Linux or MacOS
      /home/USR/.ssh/config

      USR: your username

    • Windows
      C:\USR\.ssh\config
  • Append to config

    Host snsc
      HostName 222.200.137.16
      User USR
      Port 10023
    
  • USR username

  • ssh snsc

image

Two-Factor Authentication

  • WeChat mini-program:数盾OTP

    Set your gesture

     

    Scan the QR code

    Generate the verification code

     

    Enter the verification code

     

    Enter the password

image

s2

Linux Basic Operations

image

s2 Linux Basic Operations

Help and Manual Info

Clear screen

  • ctrl + l or clear
image

Path in linux

• Absolute path

  • starts with /(root directory)

  • e.g. /public/home/xiongxn/demo

• Current working directory PWD

  • pwd or echo $PWD

• Relative path (Relative to PWD)

  • Parent directory (of PWD) .. ,
    recursively ../..

  • Current directory .

Useful abbreviations

  • home directory ~ 🢩 /public/home/USR
  • PWD 🢩 .
image

Operations

  • Change directory cd DIR

    • DIR: absolute or relative
  • List all contents in a directory ls DIR

    • DIR: absolute or relative
  • List files with pattern

    e.g. ls DIR/*.py 🢩 list all files end with .py

    * match any

  • Create an empty file: touch FILE

  • Show file content: cat FILE

  • Show first/last N lines of a file
    head -n N FILE  tail -n N FILE

  • Capture content in file
    e.g. cat make.log | grep -C 3 error

    • -C: optional, show above and below 3 lines
    • | pipe
image

Operations

  • Make directory mkdir DIR

    • DIR: absolute or relative

    • mkdir -p DIR:

  • Copy cp

    • cp FILE1 FILE2 copy FILE1 as FILE2

    • cp -r: copy dir and its contents (recursively)

  • Move mv

    • mv FILE1 FILE2 mv FILE1 as FILE2

    • mv DIR path: move DIR to path

    • mv DIR1 DIR2: rename DIR1 to DIR2
             if DIR2 dees not exist

  • Remove rm (delete)

    • remove a file rm path/FILE
    • remove a directory rm -r DIR
    • remove without confirmation (force remove)
      rm -f
image

    • Be Careful with rm -rf

    • One user account (USR) is shared by a group of participants

      You may inattentively delete files,dirs created by others

    • e.g. NEVER rm -rf ~/

      • ~/ is the shared home directory /public/home/USR

      • will remove other participants' files


    • Do not modify bashrc, bash_profile

      • Will affect other participants sharing the same user account

      • Use export ENV=VAL to set your environmental variables (usually not necessary)

image

Transfer files between remote and local

  • scp [-r] SOURCE DESTINATION

    • -r (recursively) if SOURCE is a directory
    • e.g. scp -r ./demo snsc:~/
    • Do not support resume from break-point
  • rsync [-r] SOURCE DESTINATION

    • add -v to show verbose info
    • e.g. rsync -rv ./demo snsc:~/
    • Support resume from break-point
  • Compress and Uncompress

    • tar -czvf (c for compress)
    • tar -xzvf (x for extract)
image

s2

Visual Studio Code  

image

s2

Visual Studio Code  

Download and Install

Extensions

  • Remote-SSH and Remote-Explorer
    or packed Remote-Development
  • Language support extension, e.g.

    • Python, Python Debugger, vscode-numpy-viewer
  • Indent Rainbow
    • colorize the indent in your code

    • python use indentation to identify function definition, loop structure, ...

image

Remote development using vscode

 • Use Remote-SSH extension

 • If warning popup, check Do not show agin and Allow

check Trust the authors ... folder home and Yes, I trust the authors

Enter verification code from 数盾OTP then password

Open remote directory using vscode

 

image

vscode command-line interface

  • Open directory (default in new window)

    code DIR

  • Open directory in current window

    code -r DIR

  • see code --help

image

s2

Python  
  
Numpy     matplotlib
 gvar   lsqfit

image

Python Virtual environment (venv)

  • Create: python3 -m venv VENV

    • VENV: name of your venv
  • Activate: source PATH_TO_VENV/bin/activate

  • Deactivate: deactivate

Ready-to-go python venv

  • The packages needed are already installed

  • Activate the venv source /public/home/jiangxy/venv.public/bin/activate

  • Deactivate: deactivate





image

Python REPL mode

  • Read-Evaluate-Print-Loop

    source /public/home/jiangxy/venv.public/bin/activate

image

Python script mode

  • python3 PATH_TO_SCRIPT ARGS

  • e.g. python3 ./hello.py LPC

    import sys
    
    name = sys.argv[1]
    print("hello,", name)
    
  • import: load packages

    like #include in c++

  • argv[1]: 2nd arg after python3

    in this case: LPC

    • argv counter start from 0
    • argv[0] is ./hello.py
image

Working with vscode  

Set python interpreter in vscode

  • Login snsc via vscode

  • ctrl + P, enter select interpreter

  • Choose Python: Select Interpreter

  • Choose Enter interpreter path and enter

    /public/home/jiangxy/venv.public/bin/python3

  • Now your vscode is aware of packages in venv.public and able to provide assist

image

Numpy

  • import numpy: import numpy as np

  • Core of numpy: ndarry (n-dim array)

  • Creating ndarray

    • from python list

      a = np.array([1,2,3,4,5,6])
      b = np.array([[1.,2, 3],
                    [4 ,5, 6]])
      
    • np.array(range(bgn,end))

    • np.zeros(shape,dtype)

    • np.ones(shape,dtype)

    • np.linspace(bgn,end,num)

  • Shape of np array
    a.shape

  • Vectorized mathematical functions

    np.sqrt(a), np.log(a)

  • Indexing and slicing

    a[0], a[0:4], a[1:4],a[1:-2]

    index begin with 0, but do not include the upper bound

image

Numpy

  • axis of ndarray

    • Define along which axis the operation acts

      e.g. np.sum, np.mean , np.std, ...

  • transpose ndarray

    ap = np.transpose(a,axes=(2,0,1))

    • axes: order of (axes) dims

      axes of ap 🢩 axes (2,0,1) of a

image

Numpy

  • Broadcast mechanism

    • np automatically promotes the dimensions

         


image

Numpy

Export and Load npy file

data = np.random.rand(8,6,4)
np.save("./data.npy",data)

data_p = np.load("./data.npy")

print("sum(abs(diff)) =>", 
      np.sum(np.abs(data - data_p)))
  • np.save and nup.load
extension: vscode-numpy-viewer
image

matplotlib

  • An example

    import numpy as np
    import matplotlib.pyplot as plt
    
    x_ary = np.linspace(0,4,21)
    y_ary = np.exp(-x_ary)
    plt.plot(x_ary,y_ary,'--ob')
    #plt.plot(x_ary,y_ary,ls='--', marker='o', color='blue')
    plt.ylabel('exponential')
    plt.xlabel('t')
    
    plt.savefig("exp.png")
    
image

matplotlib

  • axes and subplots

    import numpy as np
    import matplotlib.pyplot as plt
    t_ary = np.linspace(0,4,21)
    y_ary = np.exp(-t_ary)
    
    fig, (ax1,ax2) = plt.subplots(2,1,figsize=(4, 6)) 
    ax1.plot(t_ary,y_ary,'--or')
    ax2.semilogy(t_ary, y_ary)
    
    fig.savefig("subplots.png")
    
  • fig, (ax1,ax2) = plt.subplots(2,1,figsize=(4, 6))

    • plt.subplots(2, 1 🢩 2 rows × 1 columns 🢩

image

matplotlib

  • Errorbar plot

    import numpy as np
    import matplotlib.pyplot as plt
    
    t_ary = np.linspace(0,1.0,6)
    y_ary = np.array([1.,  0.82,0.67,0.54,0.45,0.37])
    y_err = np.array([0.06,0.06,0.04,0.05,0.05,0.04])
    plt.errorbar(t_ary,y_ary,y_err,ls='--', marker='o', 
                 capsize=5, capthick=1, ecolor='red')
    plt.savefig("errbar.png")
    
  • ls='--': line sstyle is dashed (--)

  • marker='o': data point is marked by • ('o')

    try other styles, e.g. -, * ...

  • capsize, capthick: , ecolor: color of errorbar
image

matplotlib anatomy

image

gvar (Gaussian Random Variables)

  • construct gvar

    >>> import gvar as gv
    >>> g = gv.gvar(1.1,0.2);
    >>> g = gv.gvar((1.1,0.2));
    >>> g = gv.gvar('1.1(0.2)');
    >>> g = gv.gvar('1.10(20)');
    >>> p = gv.gvar('1.1(0.2)e-6');
    >>> g
    1.10(20)
    

    • central value μ

      1.10(20)
            uncertainty σ
  • access gvar

    • central value: g.mean, g.val
    • standard deviation: g.sdev
    • variance: g.var
  • gvar is compilable with numpy

    np.sqrt(g), np.exp(g) ... , propagate the error

  • gv.sqrt, gv.exp are also available

  • construct an array of gvar

    g_ary=gv.gvar([1.0,1.4,2.1],[0.2,0.1,0.3])

  • mean, std of gvar array

    gv.mean(g_ary)

    gv.sdev(g_ary)

image

lsqfit (Nonlinear Least Squares Fitting)

χ2\chi^2 definition

  • Uncorrelated case χ2=∑t[Cˉt−f(t,{pi})]2σt2\dst \chi^2 = \sum_t \frac{\left[\bar{C}_t - f(t,\{p_i\})\right]^2}{\sigma_t^2}\hskip 4em

    • Cˉt\bar{C}_t 🢩 mean value of the tt-th data
    • σt \sigma_t\, 🢩 standard deviation of the tt-th data
    • No interference between different tt 🢪🢩 uncorrelated
  • Correlated case χ2=∑t,t′[Cˉt−f(t,{pi})](cov−1)t,t′[Cˉt′−f(t′,{pi})]\dst \chi^2 = \sum_{t, t'}\left[\bar{C}_{t}-f(t,\{p_i\}) \right] \left(\mathrm{cov}^{-1}\right)_{t,t'} \left[\bar{C}_{t'}-f(t',\{p_i\})\right]

    • cov\dst \mathrm{cov} : covariance matrix

      (cov)t,t′=1N−1∑i(Ct(i)−Cˉt)(Ct′(i)−Cˉt′)\left(\mathrm{cov}\right)_{t,t'}=\frac{1}{N-1}\sum_{i}(C^{(i)}_t-\bar{C}_t)(C^{(i)}_{t'}-\bar{C}_{t'})

  • See Prof. Qian Zhang's talk

image

lsqfit

  • χ2\chi^2 🢩 measure the deviation between data and model

  • least χ2\chi^2 fit 🢩 optimize {pi}\{p_i\} for minimal BUT resonable χ2\chi^2

  • lsqfit 🢩 a python package perform least χ\chi-square fit

    fit = lsqfit.nonlinear_fit(data=(var_val, dat_val), 
                               fcn=model, prior=ini_val, debug=False)
    
    • var_val: value of variables 🢩 xx-axis
    • dat_val: value of datas (with erros) 🢩 yy-axis (with err/cov)
    • model    : fit model function
    • ini_val : initial values of (also constrains on) fit parameters in model
image

lsqfit

An example

import gvar as gv
import lsqfit
import numpy as np
import matplotlib.pyplot as plt


data_ary = np.load("./data.npy") #2-d ndarray, shape 50x11
cntrl_ary = np.mean(data_ary,axis=0)
err_ary = np.std(data_ary,axis=0)
N_t = len(cntrl_ary)
t_ary = np.linspace(0,2,N_t)

t_ary_dct = {'demo': t_ary}
val_ary_dct = {'demo': gv.gvar(cntrl_ary, err_ary)}

def fit_model(t_dct, p):
    mdls = {}
    ts = t_dct['demo']
    mdls['demo'] = p['c0']*np.exp(-p['c1']*ts)
    return mdls

ini_prr = gv.gvar({'c0':'0(5)','c1':'0.2(0.5)'}) 

fit = lsqfit.nonlinear_fit(data=(t_ary_dct, val_ary_dct), \
                            fcn=fit_model, prior=ini_prr) 
print(fit.format(True))
print("\n=======\n fit.p => ", fit.p)

t_lst = np.linspace(0, 2, 50) 
fit_fcn_val = fit.fcn({'demo':t_lst}, fit.p)['demo']
fit_fcn_cntrl = gv.mean(fit_fcn_val)
fit_fcn_err = gv.sdev(fit_fcn_val)

fig, ax = plt.subplots(1,1, figsize=(10, 7))
ax.errorbar(t_ary, cntrl_ary, yerr=err_ary, fmt = 'ro', \
            alpha=0.4, label="$C_2$") 
ax.plot(t_lst, fit_fcn_cntrl, color="b", label="best fit") 
ax.fill_between(t_lst, fit_fcn_cntrl - fit_fcn_err, \
                fit_fcn_cntrl + fit_fcn_err,  alpha=0.3) 
plt.yscale("log")
plt.xlabel('t')
plt.ylabel('$C_2$')
ax.legend(loc='upper center', fontsize=10, frameon=True, \
            fancybox=True, framealpha=0.8, borderpad=0.3, \
            ncol=1, markerfirst=True, markerscale=1, \
            numpoints=1, handlelength=1.5)
fig.subplots_adjust(left=0.15, right=0.9, top=0.9, bottom=0.15)
fig.savefig("./lsqfit_demo.png")

import npy data, calculate the mean value and standard deviation, and wrap into dictionary

define the fit model

initial value (prrior)

excute fit, print results

evalute the value of best-fit model on given pointss (for plot)

plot the data, best-fit, and save to file

image

Python dictionary

t_ary_dct   = {'demo': t_ary}
val_ary_dct = {'demo': gv.gvar(cntrl_ary, err_ary)}
def fit_model(t_dct, p):
  mdls = {}
  ts = t_dct['demo']
  mdls['demo'] = p['c0']*np.exp(-p['c1']*ts)
  return mdls
ini_prr = gv.gvar({'c0': '0(5)','c1': '0.2(0.5)'}) 
  • dct = {key: value}, access the value bydct[key]

  • Why dict ?

  • Consider the case of

    • fit 2 datasets with 2 models
    • 2 models may have shared parameters
image

Combined fit

  • Combined fit: e.g. 2 data sets, 2 models (with shared parameters)

    label label1 label2
    data (x⃗(1),y⃗(1))(\vec{x}^{(1)}, \vec{y}^{(1)}) (x⃗(2),y⃗(2))(\vec{x}^{(2)}, \vec{y}^{(2)})
    model f(x,{p⃗,q⃗})f(x,\{\textcolor{blue}{\vec{p}},\vec{q}\}) g(x,{p⃗,r⃗})g(x,\{\textcolor{blue}{\vec{p}},\vec{r}\})
    ini_val p⃗0\textcolor{blue}{\vec{p}_0}, q⃗0\vec{q}_0 p⃗0,r⃗0\textcolor{blue}{\vec{p}_0}, \vec{r}_0
    • p⃗\textcolor{blue}{\vec{p}} denotes the shared parameters

See Prof. Qian Zhang's talk

image

Result of fit

print(fit.format(True))

Least Squares Fit:
chi2/dof [dof] = 0.24 [11]    Q = 0.99    logGBF = 22.832

Parameters:
           c0   0.999 (14)     [    0 ± 5.0 ]  
           c1   0.996 (25)     [  0.20 (50) ]  *

Fit:
       key        y[key]      f(p)[key]
---------------------------------------
    demo 0    1.000 (18)     0.999 (14)  
         1    0.820 (19)    0.8184 (96)  
         2    0.669 (22)    0.6705 (73)  
         3    0.549 (17)    0.5494 (67)  
         4    0.450 (20)    0.4501 (69)  
         5    0.368 (17)    0.3688 (70)  
         6    0.304 (17)    0.3021 (71)  
         7    0.246 (20)    0.2475 (69)  
         8    0.204 (24)    0.2028 (66)  
         9    0.163 (17)    0.1662 (62)  
        10    0.135 (23)    0.1361 (58)  

Settings:
  svdcut/n = 1e-12/0    tol = (1e-08*,1e-10,1e-10)    (itns/time = 7/0.1s)
  fitter = scipy_least_squares    method = trf

fig.savefig("./lsqfit_demo.png")

image

What is a good-fit ?

  • least χ2\chi^2 fit 🢩 optimize {pi}\{p_i\} for minimal BUT resonable χ2\chi^2
  • Resonable χ2/d.o.f≈1\chi^2/\mathrm{d.o.f} \approx 1
Least Square Fit:
chi2/dof [dof] = 13 [10]     
Q = 5.2e-24    
logGBF = -74.281
  • Underfit
    Model failed to discribe data
Least Square Fit:
chi2/dof [dof] = 0.07 [10]   
Q = 1    
logGBF = -46.654
  • Over fit
    Try extract too much info from data
Least Square Fit:
chi2/dof [dof] = 1 [10]      
Q = 0.41    
logGBF = -15.05
  • Reasonable fit
image

Acess the Best fit

...
fit = lsqfit.nonlinear_fit(data=(t_ary_dct, val_ary_dct), \
                          fcn=fit_model, prior=ini_prr) 
t_lst = np.linspace(0, 2, 50) 
fit_fcn_val = fit.fcn({'demo':t_lst}, fit.p)['demo']
fit_fcn_cntrl = gv.mean(fit_fcn_val)
fit_fcn_err = gv.sdev(fit_fcn_val)
...
fit.p =>  {'c0': 0.999(14), 'c1': 0.996(25)}
  • can we
c0 = fit.p['c0'] c1 = fit.p['c1'] fit_fcn_val = c0*np.exp(-c1*t_lst)

  or  

c0 = gv.gvar('0.999(14)') c1 = gv.gvar('0.996(23)') fit_fcn_val = c0*np.exp(-c1*t_lst)

  ?

try figure it out

image

s2

Slurm Workload Manager  


Environment Modules  

image


  • Simple Linux Utility for Resource Management

    • An open source, fault-tolerant, and highly scalable cluster management and job scheduling system
  • Show cluster information: sinfo

    • DO NOT run your job on login nodes

    • ALWAYS submit your job (including data analysis) to compute nodes

image

Job submit: sbatch

  • An example: ex1.sub.sh

    #!/bin/bash
    
    #SBATCH --job-name=Ex1.xnxiong   # job name
    #SBATCH --partition=cpueicc      # partition
    #SBATCH -n 1                     # totl cpu
    #SBATCH --ntasks-per-node=1      # cpu per node
    #SBATCH --output=%j.out
    #SBATCH --error=%j.err
    
    source /public/home/jiangxy/venv.public/bin/activate
    
    python3 ./Ex1.py > Ex1.log 2>&1
    

    > : redirect output stream
    2>&1: also append error message to Ex1.log

  • Submit to Slurm: sbatch ex1.sub.sh

    Submitted batch job 908048

    • 908048: job ID
image

Manage your jobs

  • Show job status: squeue --job JOB_ID

  • Terminate a job: scancel JOB_ID

    • One user account is shared by a group of participants

      • You have the privilege to cancel others' job
    • Make sure only scancel your job

      • e.g. add your name as prefix to jobname in task.sh

        #SBATCH -J xnxiong_job
        

    Ref. http://iqm.scnu.edu.cn/a/20230308/278.html

image
  • Different users may require different complier / library ...
  • How to resolve conflict/incompatible :
  • List loaded modules: module list

  • List all available modules: module avail

  • Modules needed during this summer school:

     module purge #unload all loaded modules
     module load gcc/10.3.0-gcc-4.8.5
     module load git/2.40.0-gcc-10.3.0
     module load cmake/3.22.2-gcc-10.3.0
     module load cuda/11.4.4-gcc-10.3.0
     module load hpcx/2.14/hpcx-mt
    

    put them in your job sumbmission script when needed

image

Exercise

image

1. Write a python script and submit to compute node

  • Calculate y=e−t\dst y=e^{-t}, for t=0.0,0.2,0.4,...,2t = 0.0, 0.2, 0.4, ...,2
  • Add Gaussian noise to each {ti,yi}\dst \{t_i,y_i\}
    • noise 🢩 0.02⋅ξ0.02\cdot\xi, where ξ∼N(0,1)\xi \sim N(0,1)
    • how to sample from a given distribution, check numpy.random documentation
  • Repeat Step 2 for 50 times, then you get 50 "samples"
  • Export your "samples" to .npy file

2. Write another python script and submit to compute node

  • Import previously generated .npy data file
  • Calculate the mean value and standard deviation of each yi{y_i}
  • Plot {ti,yi}\{t_i,y_i\} with error bars in linear-log plot
  • Fit the data using model f(x,{c1,c2})=c1e−c2t\dst f(x,\{c_1,c_2\}) = c_1 e^{-c_2 t}
  • Plot the best fit and data together in one figure

3. Transfer the generated figure to your local machine

Tips: numpy, np.random, np.mean, np.std, gvar, lsqfit, matplotlib, scp/rsync

image

Python venv

source /public/home/jiangxy/venv.public/bin/activate

Enviornment modules

 module purge
 module load gcc/10.3.0-gcc-4.8.5
 module load git/2.40.0-gcc-10.3.0
 module load cmake/3.22.2-gcc-10.3.0
 module load cuda/11.4.4-gcc-10.3.0
 module load hpcx/2.14/hpcx-mt

Slurm sbatch script

#!/bin/bash

#SBATCH --job-name=Ex1.xnxiong   # job name
#SBATCH --partition=cpueicc      # partition
#SBATCH -n 1                     # totl cpu
#SBATCH --ntasks-per-node=1      # cpu per node
#SBATCH --output=%j.out
#SBATCH --error=%j.err

image

Documentaion


Linux cheatsheet https://linuxopsys.com/linux-commands-cheat-sheet
numpy https://numpy.org/doc/stable/reference/
https://jalammar.github.io/visual-numpy/
matplotlib https://matplotlib.org/
https://matplotlib.org/stable/plot_types/index.html
gvar https://gvar.readthedocs.io/en/latest/index.html
lsqfit https://lsqfit.readthedocs.io/en/latest/index.html